A.1 MONet To segment each w hframe Ft into No object representations, MONet uses a recurrent attention network to obtain No attention masks Ati [0,1]w h for i = 1,...,No that represent the probability of each pixel in Ft belonging to the i-th object, with This attention network is coupled with a component VAE with latents zti Rd for i= 1,...,No that reconstructs Ati Ft, the i-th object in the image. The latent posterior distribution q(zt|Ft,Ati)is a diagonal Gaussian with mean µti, and we use µti as the representation of the i-th object. When these representations are fed into the transformer, we use a linear projection to map the raw object/word embeddings, which lie in Rd, to a vector in RdNH, where NH is the number of selfattention heads. This step is necessary as generally the latent dimensionality of MONet, d, is less than NH whereas a transformer expects the embedding size to be divisible by NH. A.2 Self-supervised training Recall in the main text that we wrote the auxiliary self-supervised loss as auxiliary loss = X A comparison of these losses and the masking schemes is given in Figure 4. We also tested a few variations of the contrastive loss inspired by literature and tested all combinations of variations.